Calibration transfer via filter learning.

BACKGROUND: Calibration transfer is an essential activity in analytical chemistry in order to avoid a complete recalibration. Currently, the most popular calibration transfer methods, such as piecewise direct standardization and dynamic orthogonal projection, require a certain amount of standard or reference samples to guarantee their effectiveness. To achieve higher efficiency, it is desirable to perform the transfer with as few reference samples as possible. RESULTS: To this end, we propose a new calibration transfer method by using a calibration database from a master instrument (source domain) and only one spectrum with known properties from a slave instrument (target domain). We first generate a counterpart of this spectrum in the source domain by a multivariate Gaussian kernel. Then, we train a filter to make the response function of the slave instrument equivalent to that of the master instrument. To avoid the need for labels from the target domain, we also propose an unsupervised way to implement our method. Compared with several state-of-the-art methods, the results on one simulated dataset and two real-world datasets demonstrate the effectiveness of our method. SIGNIFICANCE: Traditionally, the demand for certain amounts of reference samples during calibration transfer is cumbersome. Our approach, which requires only one reference sample, makes the transfer process simple and fast. In addition, we provide an alternative for performing unsupervised calibration transfer. As such, the proposed method is a promising tool for calibration transfer.

Visible - Near infrared hyperspectral dataset of healthy and infected apple tree leaves images for the monitoring of apple fire blight.

This dataset consists of three groups of hyperspectral images of apple tree plants. The first group of images consists of a temporal monitoring of seven apple tree plants, infected with fire blight (Erwinia amylovora), and six control plants over a period of 15 days. The second group of images includes a temporal monitoring of three infected plants, seven plants subjected to water stress, and seven control plants. The third group of images corresponds to acquisitions made in the orchard on nine trees showing symptoms of fire blight and six control trees. The pixel locations of infected areas have been provided for all images featuring symptomatic plants.

Monitoring of water sorption and swelling of potato starch-glycerol extruded blend by magnetic resonance imaging and multivariate curve resolution.

Magnetic resonance microimaging (MRµI) is an outstanding technique for studying water transfers in millimetric bio-based materials in a non-destructive and non-invasive manner. However, depending on the composition of the material, monitoring and quantification of these transfers can be very complex, and hence reliable image processing and analysis tools are necessary. In this study, a combination of MRµI and multivariate curve resolution-alternating least squares (MCR-ALS) is proposed to monitor the water ingress into a potato starch extruded blend containing 20% glycerol that was shown to have interesting properties for biomedical, textile, and food applications. In this work, the main purpose of MCR is to provide spectral signatures and distribution maps of the components involved in the water uptake process that occurs over time with various kinetics. This approach allowed the description of the system evolution at a global (image) and a local (pixel) level, hence, permitted the resolution of two waterfronts, at two different times into the blend that could not be resolved by any other mathematical processing method usually used in magnetic resonance imaging (MRI). The results were supplemented by scanning electron microscopy (SEM) observations in order to interpret these two waterfronts in a biological and physico-chemical point of view.

Fast and robust NIRS-based characterization of raw organic waste: Using non-linear methods to handle water effects.

Fast characterization of organic waste using near infrared spectroscopy (NIRS) has been successfully developed in the last decade. However, up to now, an on-site use of this technology has been hindered by necessary sample preparation steps (freeze-drying and grinding) to avoid important water effects on NIRS. Recent research studies have shown that these effects are highly non-linear and relate both to the biochemical and physical properties of samples. To account for these complex effects, the current study compares the use of many different types of non-linear methods such as partial least squares regression (PLSR) based methods (global, clustered and local versions of PLSR), machine learning methods (support vector machines, regression trees and ensemble methods) and deep learning methods (artificial and convolutional neural networks). On an independent test data set, non-linear methods showed errors 28% lower than linear methods. The standard errors of prediction obtained for the prediction of total solids content (TS%), chemical oxygen demand (COD) and biochemical methane potential (BMP) were respectively 8%, 160 mg(O2).gTS-1 and 92 mL(CH4).gTS-1. These latter errors are similar to successful NIRS applications developed on freeze-dried samples. These findings hold great promises regarding the development of at-site and online NIRS solutions in anaerobic digestion plants.

Preprocessing NIR Spectra for Aquaphotomics.

Even though NIR spectroscopy is based on the Beer-Lambert law, which clearly relates the concentration of the absorbing elements with the absorbance, the measured spectra are subject to spurious signals, such as additive and multiplicative effects. The use of NIR spectra, therefore, requires a preprocessing step. This article reviews the main preprocessing methods in the light of aquaphotomics. Simple methods for visualizing the spectra are proposed in order to guide the user in the choice of the best preprocessing. The most common chemometrics preprocessing are presented and illustrated by three real datasets. Some preprocessing aims to produce a spectrum as close as possible to the absorbance that would have been measured under ideal conditions and is very useful for the establishment of an aquagram. Others, dedicated to the improvement of the resolution of the spectra, are very useful for the identification of the peaks. Finally, special attention is given to the problem of reducing multiplicative effects and to the potential pitfalls of some very popular methods in chemometrics. Alternatives proposed in recent papers are presented.

N-CovSel, a new strategy for feature selection in N-way data.

In data analysis, how to select meaningful variables is a hot and wide-debated topic, and several variable selection (or feature reduction) approaches have been proposed in the literature. Although feature selection methods are numerous, most of them are suitable for data matrices, but not for higher order structures. This is mainly due to the fact the assessment of the relevancy of variables in a multi-way context has not been extensively discussed. To the best of our knowledge, among variable selection approaches developed for standard 2-way data arrays, only VIP analysis and selectivity ratio have been extended to higher-order structures. This aspect is not given by an irrelevance of the topic; on the contrary, the possibility of selecting information in a complex data set such as a multi-way structure is crucial. In the light of these considerations, the present paper discusses a feature selection strategy for N-way data based on the Covariance Selection (CovSel) approach, thus called N-CovSel. This method allows the selection of features of different dimensionality (from 1- up to (N-1)-way), depending on the nature of the original data array. The novel method has been applied on a simulated data set, in order to inspect its ability in selecting features compatible with the ground truth of the system, and on a real data set. In both cases, N-CovSel has demonstrated to be able to select meaningful features. Eventually, different strategies for the further analysis of the selected features have been proposed; some, based on sequential multi-block methods, providing a further data reduction, and some, N-PLS-based, respecting the multi-way nature of the data.

Unsupervised dynamic orthogonal projection. An efficient approach to calibration transfer without standard samples.

Calibration transfer has been traditionally performed in the context of transferring models between instruments using standard samples. Recently, new methodologies and applications have shown that transfer techniques can be adopted to achieve calibration transfer between other types of domains, such as product form, variant or seasonality. In addition, to achieving a higher efficiency for calibration transfer, it is desirable to perform the transfer without the need for standard samples or new reference analyses. Therefore, we propose a method for unsupervised calibration transfer based on the orthogonalization for structural differences between domains. The method has been successfully applied to one simulated dataset and two real datasets. In the studied cases, the proposed methodology allowed to achieve a successful transfer of calibration models and enabled the interpretation of the interferences responsible for the degradation of the original calibration models when transferred to the new domain.

Multiblock Analysis Applied to Fluorescence and Absorbance Spectra to Estimate Total Polyphenol Content in Extra Virgin Olive Oil.

A fast and easy methodology to estimate total polyphenol content in extra virgin olive oil was developed by applying the chemometric multiblock method sequential and orthogonalized partial least squares (SO-PLS) in order to combine front-face emission fluorescence spectra (270 nm excitation wavelength) and absorbance spectra. The hypothesis of this work stated that inner-filter effects in fluorescence spectra that would reduce the estimation performance of a single block model could be overcome by incorporating the absorbance spectral information of the compounds causing them. Different spectral preprocessing algorithms were applied. Double cross-validation with 50 iterations was implemented to improve the robustness of the obtained results. The PLSR model on the single block of fluorescence raw spectra achieved an RMSEP of 177.11 mg·kg-1 as the median value, and the complexity of the model was high, as the median value of latent variables (LVs) was eight. Multiblock SO-PLS models with pretreated fluorescence and absorbance spectra provided better performance, although artefacts could be introduced by transformation. The combination of fluorescence and absorbance raw data decreased the RMSEP median to 134.45 mg·kg-1. Moreover, the complexity of the model was greatly reduced, which contributed to an increase in robustness. The median value of LVs was three for fluorescence data and only one for absorbance data. Validation of the methodology could be addressed by further work considering a higher number of samples and a detailed composition of polyphenols.

A novel robust PLS regression method inspired from boosting principles: RoBoost-PLSR.

The calibration of Partial Least Square regression (PLSR) models can be disturbed by outlying samples in the data. In these cases the models can be unstable and their predictive potential can be depreciated. To address this problem, some robust versions of the PLSR Algorithm were proposed. These algorithms rely on the downweighting of these outliers during calibration. To this end, it is necessary to estimate an inconsistency measurement between the samples and the model. However, this estimation is not trivial in high dimensions. This paper proposes a novel robust PLSR algorithm inspired from the principles of boosting: RoBoost-PLSR. This method consists of realising a series of one latent variable weighted PLSR. RoBoost-PLSR is compared with the PLSR algorithm calibrated with and without outliers and also with Partial Robust M-regression (PRM), a reference robust method. This evaluation is conducted on the basis of three simulated datasets and a real dataset. Finally Roboost-PLSR proves to be resilient to the tested outliers, and can achieve the performances of the reference PLSR calibrated without any outlier.

Untargeted analysis of TD-NMR signals using a multivariate curve resolution approach: Application to the water-imbibition kinetics of Arabidopsis seeds.

The aim of this study is to investigate the ability of Time-Domain Nuclear Magnetic Resonance (TD-NMR) combined with Multivariate Curve Resolution Alternating Least Squares (MCR-ALS) analysis to detect changes in hydration properties of nineteen genotypes of Arabidopsis (Arabidopsis thaliana) seeds during the imbibition process. The Hybrid hard and Soft modelling version of MCR-ALS (HS-MCR) applied to raw TD-NMR data allowed the introduction of kinetic models to elucidate underlying biological mechanisms. The imbibition process of all investigated hydrated Arabidopsis seeds could be described with a kinetic model based on two consecutive first-order reactions related to an initial absorption of water from the bulk around the seed and a posteriori hydration of the internal seed tissues, respectively. Good data fit was achieved (LOF % = 0.98 and r2% = 99.9), indicating that the hypothesis of the selected kinetic model was correct. An interpretation of the mucilage characteristics of the studied Arabidopsis seeds was also provided. The presented methodology offers a novel and general strategy to describe in a comprehensive way the kinetic process of plant tissue hydration in a screening objective. This work also proves the potential of the MCR methods to analyse raw TD-NMR signals as alternative to the controversial and time-consuming pre-processing techniques of this kind of data, known to be an ill-conditioned and ill-posed problem.

On-site substrate characterization in the anaerobic digestion context: A dataset of near infrared spectra acquired with four different optical systems on freeze-dried and ground organic waste.

The near infrared spectra of thirty-three freeze-dried and ground organic waste samples of various biochemical composition were collected on four different optical systems, including a laboratory spectrometer, a transportable spectrometer with two measurement configurations (an immersed probe, and a polarized light system) and a micro-spectrometer. The provided data contains one file per spectroscopic system including the reflectance or absorbance spectra with the corresponding sample name and wavelengths. A reference data file containing carbohydrates, lipid and nitrogen content, biochemical methane potential (BMP) and chemical oxygen demand (COD) for each sample is also provided. This data enables the comparison of the optical systems for predictive model calibration based for example on Partial Least Squares Regression (PLS-R) [1], but could be used more broadly to test new chemometrics methods. For example, the data could be used to evaluate different transfer functions between spectroscopic systems [2]. This dataset enabled the research work reported by Mallet et al. 2021 [3].

Fast at-line characterization of solid organic waste: Comparing analytical performance of different compact near infrared spectroscopic systems with different measurement configurations.

Fast characterization of solid organic waste using near infrared spectroscopy has been successfully developed in the last decade. However, its adoption in biogas plants for monitoring the feeding substrates remains limited due to the lack of applicability and high costs. Recent evolutions in the technology have given rise to both more compact and more modular low-cost near infrared systems which could allow a larger scale deployment. The current study investigates the relevance of these new systems by evaluating four different Fourier transform near-infrared spectroscopic systems with different compactness (laboratory, portable, micro spectrometer) but also different measurement configurations (polarized light, at distance, in contact). Though the conventional laboratory spectrometer showed the best performance on the various biochemical parameters tested (carbohydrates, lipids, nitrogen, chemical oxygen demand, biochemical methane potential), the compact systems provided very close results. Prediction of the biochemical methane potential was possible using a low-cost micro spectrometer with an independent validation set error of only 91 NmL(CH4).gTS-1 compared to 60 NmL(CH4).gTS-1 for a laboratory spectrometer. The differences in performance were shown to result mainly from poorer spectral sampling; and not from instrument characteristics such as spectral resolution. Regarding the measurement configurations, none of the evaluated systems allowed a significant gain in robustness. In particular, the polarized light system provided better results when using its multi-scattered signal which brings further evidence of the importance of physical light-scattering properties in the success of models built on solid organic waste.

Chemometric pre-processing can negatively affect the performance of near-infrared spectroscopy models for fruit quality prediction.

Chemometrics pre-processing of spectral data is widely performed to enhance the predictive performance of near-infrared (NIR) models related to fresh fruit quality. Pre-processing approaches in the domain of NIR data analysis are used to remove the scattering effects, thus, enhancing the absorption components related to the chemical properties. However, in the case of fresh fruit, both the scattering and absorption properties are of key interest as they jointly explain the physicochemical state of a fruit. Therefore, pre-processing data that reduces the scattering information in the spectra may lead to poorly performing models. The objectives of this study are to test two hypotheses to explore the effect of pre-processing on NIR spectra of fresh fruit. The first hypothesis is that the pre-processing of NIR spectra with scatter correction techniques can reduce the predictive performance of models as the scatter correction can reduce the useful scattering information correlated to the property of interest. The second hypothesis is that the Deep Learning (DL) can model the raw absorbance data (mix of scattering and absorption) much more efficiently than the Partial Least Squares (PLS) regression analysis. To test the hypotheses, a real NIR data set related to dry matter (DM) prediction in mango fruit was used. The dataset consisted of a total of 11,420 NIR spectra and reference DM measurements for model training and independent testing. The chemometric pre-processing methods explored were standard normal variate (SNV), variable sorting for normalization (VSN), Savitzky-Golay based 2nd derivative and their combinations. Further two modelling approaches i.e., PLS regression and DL were used to evaluate the effect of pre-processing. The results showed that the best root mean squared error of prediction (RMSEP) for both the PLS and DL models were obtained with the raw absorbance data. The spectral pre-processing in general decreased the performance of both the PLS and DL models. Further, the DL model attained the lowest RMSEP of 0.76%, which was 13% lower compared to the PLS regression on the raw absorbance data. Pre-processing approaches should be carefully used while analysing the NIR data related to fresh fruit.

Relating Near-Infrared Light Path-Length Modifications to the Water Content of Scattering Media in Near-Infrared Spectroscopy: Toward a New Bouguer-Beer-Lambert Law.

In near-infrared spectroscopy (NIRS), the linear relationship between absorbance and an absorbing compound concentration has been strictly defined by the Bouguer-Beer-Lambert law only for the case of transmission measurements of nonscattering media. However, various quantitative calibrations have been successfully built both on reflectance measurements and for scattering media. Although the lack of linearity for scattering media has been observed experimentally, the sound multivariate statistics and signal processing involved in chemometrics have allowed us to overcome this problem in most cases. However, in the case of samples with varying water content, important modifications of scattering levels still make calibrations difficult to build due to nonlinearities. Moreover, even when calibration procedures are successfully developed, many preprocessing methods used do not guarantee correct spectroscopic assignments (in the sense of a pure chemical absorbance). In particular, this may prevent correct modeling and interpretation of the structure of water. In this study, dynamic near-infrared spectra acquired during a drying process allow the study of the physical effects of water content variations, with a focus on the first overtone OH absorbance region. A model sample consisting of aluminum pellets mixed with water allowed us to study this specifically, without any other absorbing interaction terms related to the dry mass-absorbing constituents. A new formulation of the Bouguer-Beer-Lambert law is proposed, by expressing path length as a power function of water content. Through this new formulation, it is shown that a better and simpler prediction model of water content may be developed, with more precise and accurate identification of water absorbance bands.

Unveiling non-linear water effects in near infrared spectroscopy: A study on organic wastes during drying using chemometrics.

In the context of organic waste management, near infrared spectroscopy (NIRS) is being used to offer a fast, non-destructive, and cost-effective characterization system. However, cumbersome freeze-drying steps of the samples are required to avoid water's interference on near infrared spectra. In order to better understand these effects, spectral variations induced by dry matter content variations were obtained for a wide variety of organic substrates. This was made possible by the development of a customized near infrared acquisition system with dynamic highly-resolved simultaneous scanning of near infrared spectra and estimation of dry matter content during a drying process at ambient temperature. Using principal components analysis, the complex water effects on near infrared spectra are detailed. Water effects are shown to be a combination of both physical and chemical effects, and depend on both the characteristics of the samples (biochemical type and physical structure) and the moisture content level. This results in a non-linear relationship between the measured signal and the analytical characteristic of interest. A typology of substrates with respect to these water effects is provided and could further be efficiently used as a basis for the development of local quantitative calibration models and correction methods accounting for these water effects.

Sequential fusion of information from two portable spectrometers for improved prediction of moisture and soluble solids content in pear fruit.

Near infrared (NIR) spectroscopy allows rapid estimation of quality traits in fresh fruit. Several portable spectrometers are available in the market as a low-cost solution to perform NIR spectroscopy. However, portable spectrometers, being lower in cost than a benchtop counterpart, do not cover the complete near infrared (NIR) spectral range. Often portable sensors either use silicon-based visible and NIR detector to cover 400-1000 nm, or InGaAs-based short wave infrared (SWIR) detector covering the 900-1700 nm. However, these two spectral regions carry complementary information, since the 400-1000 nm interval captures the color and 3rd overtones of most functional group vibrations, while the 1st and the 2nd overtones of the same transitions fall in the 1000-1700 nm range. To exploit such complementarity, sequential data fusion strategies were used to fuse the data from two portable spectrometers, i.e., Felix F750 (~400-1000 nm) and the DLP NIR Scan Nano (~900-1700 nm). In particular, two different sequential fusion approaches were used, namely sequential orthogonalized partial-least squares (SO-PLS) regression and sequential orthogonalized covariate selection (SO-CovSel). SO-PLS improved the prediction of moisture content (MC) and soluble solids content (SSC) in pear fruit, leading to an accuracy which was not obtainable with models built on any of the two spectral data set individually. Instead, SO-CovSel was used to select the key wavelengths from both the spectral ranges mostly correlated to quality parameters of pear fruit. Sequential fusion of the data from the two portable spectrometers led to an improved model prediction (higher R2 and lower RMSEP) of MC and SSC in pear fruit: compared to the models built with the DLP NIR Scan Nano (the worst individual block) where SO-PLS showed an increase in R2p up to 56% and a corresponding 47% decrease in RMSEP. Differences were less pronounced to the use of Felix data alone, but still the R2p was increased by 2.5% and the RMSEP was reduced by 6.5%. Sequential data fusion is not limited to NIR data but it can be considered as a general tool for integrating information from multiple sensors.

Multi-block classification of chocolate and cocoa samples into sensory poles.

The present study aims at developing an analytical methodology which allows correlating sensory poles of chocolate to their chemical characteristics and, eventually, to those of the cocoa beans used for its preparation. Trained panelists investigated several samples of chocolate, and they divided them into four sensorial poles (characterized by 36 different descriptors) attributable to chocolate flavor. The same samples were analyzed by six different techniques: Proton Transfer Reaction-Time of Flight-Mass Spectrometry (PTR-ToF-MS), Solid Phase Micro Extraction-Gas Chromatography-Mass Spectroscopy (SPME-GC-MS), High-Performance Liquid Chromatography (HPLC) (for the quantification of eight organic acids), Ultra High Performance Liquid Chromatography coupled to triple-quadrupole Mass Spectrometry (UHPLC-QqQ-MS) for polyphenol quantification, 3D front face fluorescence Spectroscopy and Near Infrared Spectroscopy (NIRS). A multi-block classification approach (Sequential and Orthogonalized-Partial Least Squares - SO-PLS) has been used, in order to exploit the chemical information to predict the sensorial poles of samples. Among thirty-one test samples, only two were misclassified.

Improved prediction of tablet properties with near-infrared spectroscopy by a fusion of scatter correction techniques.

Near-infrared (NIR) spectra of pharmaceutical tablets get affected by light scattering phenomena, which mask the underlying peaks related to chemical components. Often the best performing scatter correction technique is selected from a pool of pre-selected techniques. However, the data corrected with different techniques may carry complementary information, hence, use of a single scatter correction technique is sub-optimal. In this study, the aim is to prove that NIR models related to pharmaceuticals can directly benefit from the fusion of complementary information extracted from multiple scatter correction techniques. To perform the fusion, sequential and parallel pre-processing fusion approaches were used. Two different open source NIR data sets were used for the demonstration where the assay uniformity and active ingredient (AI) content prediction was the aim. As a baseline, the fusion approach was compared to partial least-squares regression (PLSR) performed on standard normal variate (SNV) corrected data, which is a commonly used scatter correction technique. The results suggest that multiple scatter correction techniques extract complementary information and their complementary fusion is essential to obtain high-performance predictive models. In this study, the prediction error and bias were reduced by up to 15 % and 57 % respectively, compared to PLSR performed on SNV corrected data.

Improved prediction of fuel properties with near-infrared spectroscopy using a complementary sequential fusion of scatter correction techniques.

Near-infrared (NIR) spectroscopy of fuels can suffer from scattering effects which may mask the signals corresponding to key analytes in the spectra. Therefore, scatter correction techniques are often used prior to any modelling so to remove scattering and improve predictive performances. However, different scatter correction techniques may carry complementary information so that, if jointly used, both model stability and performances could be improved. A solution to that is the fusion of complementary information from differently scatter corrected data. In the present work, the use of a preprocessing fusion approach called sequential preprocessing through orthogonalization (SPORT) is demonstrated for predicting key quality parameters in diesel fuels. In particular, the possibility of predicting four different key properties, i.e., boiling point (°C), density (g/mL), aromatic mass (%) and viscosity (cSt), was considered. As a comparison, standard partial least-squares (PLS) regression modelling was performed on data pretreated by SNV and 2nd derivative (which is a widely used preprocessing combination). The results showed that the SPORT models, based on the fusion of scatter correction techniques, outperformed the standard PLS models in the prediction of all the four properties, suggesting that selection and use of a single scatter correction technique is often not sufficient. Up to complete bias removal with 50% reduction in prediction error was obtained. The R2P was increased by up to 8%. The sequential scatter fusion approach (SPORT) is not limited to NIR data but can be applied to any other spectral data where a preprocessing optimization step is required.

Visible and Near-Infrared Multispectral Features in Conjunction with Artificial Neural Network and Partial Least Squares for Predicting Biochemical and Micro-Structural Features of Beef Muscles.

The objective of this study was to determine the potential of multispectral imaging (MSI) data recorded in the visible and near infrared electromagnetic regions to predict the structural features of intramuscular connective tissue, the proportion of intramuscular fat (IMF), and some characteristic parameters of muscle fibers involved in beef sensory quality. In order to do this, samples from three muscles (Longissimus thoracis, Semimembranosus and Biceps femoris) of animals belonging to three breeds (Aberdeen Angus, Limousine, and Blonde d'Aquitaine) were used (120 samples). After the acquisition of images by MSI and segmentation of their morphological parameters, a back propagation artificial neural network (ANN) model coupled with partial least squares was applied to predict the muscular parameters cited above. The results presented a high accuracy and are promising (R2 test > 0.90) for practical applications. For example, considering the prediction of IMF, the regression model giving the best ANN model exhibited R2P = 0.99 and RMSEP = 0.103 g × 100 g-1 DM.